[Segment Replication] Refactor RemoteStoreReplicationSource (#8767) #9076

dreamer-89 · 2023-08-02T18:20:46Z

Manual backport of #8767 to 2.x

Note: This needs to be merged in after as this PR contain changes on top of below two PRs. Keeping in draft until below are backported.

github-actions · 2023-08-02T18:31:09Z

Gradle Check (Jenkins) Run Completed with:

RESULT: FAILURE ❌
URL: https://build.ci.opensearch.org/job/gradle-check/21715/
CommitID: 2e37b6b
Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green.
Is the failure a flaky test unrelated to your change?

…ch-project#8767) * [Segment Replication] Refactor remote replication source Signed-off-by: Suraj Singh <[email protected]> * Unit test updates Signed-off-by: Suraj Singh <[email protected]> * Self review Signed-off-by: Suraj Singh <[email protected]> * Self review Signed-off-by: Suraj Singh <[email protected]> * Segregate shard level tests for node to node and remote store segment replication Signed-off-by: Suraj Singh <[email protected]> * Fix failing unit tests Signed-off-by: Suraj Singh <[email protected]> * Fix failing UT Signed-off-by: Suraj Singh <[email protected]> * Fix failing UT Signed-off-by: Suraj Singh <[email protected]> * Address review comments Signed-off-by: Suraj Singh <[email protected]> * Fix more unit tests Signed-off-by: Suraj Singh <[email protected]> * Improve RemoteStoreReplicationSourceTests, remove unnecessary mocks and use actual failures for failure/exception use cases Signed-off-by: Suraj Singh <[email protected]> * Spotless check fix Signed-off-by: Suraj Singh <[email protected]> * Address review comments Signed-off-by: Suraj Singh <[email protected]> * Ignore files already in store while computing segment file diff with primary Signed-off-by: Suraj Singh <[email protected]> * Spotless fix Signed-off-by: Suraj Singh <[email protected]> * Fix failing UT Signed-off-by: Suraj Singh <[email protected]> * Spotless fix Signed-off-by: Suraj Singh <[email protected]> * Move read/writes from IndexInput/Output to RemoteSegmentMetadata Signed-off-by: Suraj Singh <[email protected]> * Address review commnt Signed-off-by: Suraj Singh <[email protected]> * Update recovery flow to perform commits during recovery Signed-off-by: Suraj Singh <[email protected]> * Remove un-necessary char Signed-off-by: Suraj Singh <[email protected]> * Address review comments Signed-off-by: Suraj Singh <[email protected]> * Update comment nit-pick Signed-off-by: Suraj Singh <[email protected]> * Remove deletion logic causing read issues due to deleted segments_N Signed-off-by: Suraj Singh <[email protected]> * Spotless fix Signed-off-by: Suraj Singh <[email protected]> * Fix unit tests Signed-off-by: Suraj Singh <[email protected]> --------- Signed-off-by: Suraj Singh <[email protected]>

Signed-off-by: Suraj Singh <[email protected]>

github-actions · 2023-08-03T17:21:39Z

Gradle Check (Jenkins) Run Completed with:

RESULT: FAILURE ❌
URL: https://build.ci.opensearch.org/job/gradle-check/21800/
CommitID: fbed447
Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green.
Is the failure a flaky test unrelated to your change?

Signed-off-by: Suraj Singh <[email protected]>

opensearch-trigger-bot · 2023-08-03T17:42:33Z

Compatibility status:



> Task :checkCompatibility
Incompatible components: [https://github.com/opensearch-project/geospatial.git, https://github.com/opensearch-project/index-management.git, https://github.com/opensearch-project/notifications.git, https://github.com/opensearch-project/k-nn.git, https://github.com/opensearch-project/alerting.git, https://github.com/opensearch-project/anomaly-detection.git, https://github.com/opensearch-project/cross-cluster-replication.git, https://github.com/opensearch-project/ml-commons.git]
Components skipped due to git failures: [https://github.com/opensearch-project/opensearch-oci-object-storage.git]
Compatible components: [https://github.com/opensearch-project/security.git, https://github.com/opensearch-project/job-scheduler.git, https://github.com/opensearch-project/neural-search.git, https://github.com/opensearch-project/security-analytics.git, https://github.com/opensearch-project/sql.git, https://github.com/opensearch-project/observability.git, https://github.com/opensearch-project/asynchronous-search.git, https://github.com/opensearch-project/common-utils.git, https://github.com/opensearch-project/performance-analyzer.git, https://github.com/opensearch-project/reporting.git, https://github.com/opensearch-project/performance-analyzer-rca.git]

BUILD SUCCESSFUL in 31m 26s

opensearch-trigger-bot · 2023-08-03T17:57:54Z

Compatibility status:



> Task :checkCompatibility
Incompatible components: [https://github.com/opensearch-project/geospatial.git, https://github.com/opensearch-project/index-management.git, https://github.com/opensearch-project/notifications.git, https://github.com/opensearch-project/k-nn.git, https://github.com/opensearch-project/alerting.git, https://github.com/opensearch-project/anomaly-detection.git, https://github.com/opensearch-project/cross-cluster-replication.git, https://github.com/opensearch-project/ml-commons.git]
Components skipped due to git failures: [https://github.com/opensearch-project/opensearch-oci-object-storage.git]
Compatible components: [https://github.com/opensearch-project/security.git, https://github.com/opensearch-project/neural-search.git, https://github.com/opensearch-project/job-scheduler.git, https://github.com/opensearch-project/security-analytics.git, https://github.com/opensearch-project/sql.git, https://github.com/opensearch-project/observability.git, https://github.com/opensearch-project/asynchronous-search.git, https://github.com/opensearch-project/performance-analyzer.git, https://github.com/opensearch-project/common-utils.git, https://github.com/opensearch-project/performance-analyzer-rca.git, https://github.com/opensearch-project/reporting.git]

BUILD SUCCESSFUL in 29m 57s

github-actions · 2023-08-03T18:10:14Z

Gradle Check (Jenkins) Run Completed with:

RESULT: UNSTABLE ❕
TEST FAILURES:

      1 org.opensearch.client.PitIT.testDeleteAllAndListAllPits

URL: https://build.ci.opensearch.org/job/gradle-check/21802/
CommitID: 0e9c7d7
Please review all flaky tests that succeeded after retry and create an issue if one does not already exist to track the flaky failure.

codecov · 2023-08-03T18:12:48Z

Codecov Report

Merging #9076 (0e9c7d7) into 2.x (6ab46bd) will increase coverage by 0.13%.
The diff coverage is 97.53%.

@@             Coverage Diff              @@
##                2.x    #9076      +/-   ##
============================================
+ Coverage     70.72%   70.86%   +0.13%     
- Complexity    57372    57497     +125     
============================================
  Files          4754     4754              
  Lines        271625   271647      +22     
  Branches      40066    40067       +1     
============================================
+ Hits         192102   192494     +392     
+ Misses        62972    62655     -317     
+ Partials      16551    16498      -53

Files Changed	Coverage Δ
...ch/indices/recovery/PeerRecoveryTargetService.java	`51.96% <0.00%> (-1.97%)`	⬇️
...ices/replication/RemoteStoreReplicationSource.java	`90.74% <96.42%> (-3.55%)`	⬇️
...in/java/org/opensearch/index/shard/IndexShard.java	`69.11% <100.00%> (-0.59%)`	⬇️
...search/index/shard/RemoteStoreRefreshListener.java	`84.61% <100.00%> (-0.09%)`	⬇️
...java/org/opensearch/index/shard/StoreRecovery.java	`57.25% <100.00%> (ø)`
...earch/index/store/RemoteSegmentStoreDirectory.java	`87.70% <100.00%> (ø)`
...rc/main/java/org/opensearch/index/store/Store.java	`80.65% <100.00%> (+0.40%)`	⬆️
...x/store/remote/metadata/RemoteSegmentMetadata.java	`100.00% <100.00%> (ø)`
...h/indices/replication/GetSegmentFilesResponse.java	`44.44% <100.00%> (+6.94%)`	⬆️
.../indices/replication/SegmentReplicationTarget.java	`89.32% <100.00%> (+5.48%)`	⬆️

... and 473 files with indirect coverage changes

dreamer-89 added the skip-changelog label Aug 2, 2023

dreamer-89 marked this pull request as draft August 2, 2023 18:21

dreamer-89 added 2 commits August 3, 2023 10:06

Rebase 2.x and resolve conflicts

fbed447

Signed-off-by: Suraj Singh <[email protected]>

dreamer-89 force-pushed the backport/backport-8767-to-2.x branch from 2e37b6b to fbed447 Compare August 3, 2023 17:10

dreamer-89 marked this pull request as ready for review August 3, 2023 17:11

mch2 approved these changes Aug 3, 2023

View reviewed changes

Fix unit test

0e9c7d7

Signed-off-by: Suraj Singh <[email protected]>

dreamer-89 mentioned this pull request Aug 3, 2023

Make sure IndexOutput of metadata file is always closed #8653

Merged

6 tasks

Poojita-Raj approved these changes Aug 3, 2023

View reviewed changes

dreamer-89 merged commit 94338e5 into opensearch-project:2.x Aug 3, 2023

dreamer-89 deleted the backport/backport-8767-to-2.x branch August 3, 2023 18:14

dreamer-89 mentioned this pull request Aug 3, 2023

[AUTOCUT] Gradle Check Failure on push to 2.x #9096

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[Segment Replication] Refactor RemoteStoreReplicationSource (#8767) #9076

[Segment Replication] Refactor RemoteStoreReplicationSource (#8767) #9076

dreamer-89 commented Aug 2, 2023 •

edited

Loading

github-actions bot commented Aug 2, 2023

github-actions bot commented Aug 3, 2023

opensearch-trigger-bot bot commented Aug 3, 2023

opensearch-trigger-bot bot commented Aug 3, 2023

github-actions bot commented Aug 3, 2023

codecov bot commented Aug 3, 2023

[Segment Replication] Refactor RemoteStoreReplicationSource (#8767) #9076

[Segment Replication] Refactor RemoteStoreReplicationSource (#8767) #9076

Conversation

dreamer-89 commented Aug 2, 2023 • edited Loading

github-actions bot commented Aug 2, 2023

Gradle Check (Jenkins) Run Completed with:

github-actions bot commented Aug 3, 2023

Gradle Check (Jenkins) Run Completed with:

opensearch-trigger-bot bot commented Aug 3, 2023

opensearch-trigger-bot bot commented Aug 3, 2023

github-actions bot commented Aug 3, 2023

Gradle Check (Jenkins) Run Completed with:

codecov bot commented Aug 3, 2023

Codecov Report

dreamer-89 commented Aug 2, 2023 •

edited

Loading